This invention relates to a system and method for transcribing and editing freeform image data using a structured freeform editor. More particularly, the invention is directed to a system and method for transcribing and editing freeform graphic elements (FGE(s)) such as scribbles and other graphical objects associated with scribble input devices (such as interactive display boards), scanned images, graphic programs, pen-based ink formats and raster-converted graphics. The subject editor can be used for both transcription from freeform graphic elements (such as scribble) to an editable text format and for editing of the interpreted structure of the freeform graphic elements. The editor performs basic functions that appear to the user to be simple. However, the output of the editor is important because it may then be integrated with other high end editors for situations where greater control of the output format is required.
While the invention is particularly directed to the art of transcribing and editing data representing freeform graphic elements such as scribbles, text and other graphical objects displayed on a scribble device or otherwise input into the system, and will thus be described with specific reference thereto, it will be appreciated that the invention may have usefulness in other fields and applications.
By way of background, it is generally accepted that the construction of textual matter is important for most computer applications. In the early development of computers, only plain-text editors were implemented on large computers. These editors included basic functions such as line formatting, tabbing, capitalization, etc. and other functions typically available on typewriters. Soon after the introduction of the low-cost laser printer by Apple in 1984, however, a new type of text and graphic application programs emerged, referred to here as “structured editors”, which dealt with text format and layout, document structures, page layout, and graphic diagrams. Examples of such structured editors include Microsoft Word (for text documents), Adobe PageMill (for internet web pages), Microsoft Excel (for spreadsheets), and Visio (for diagrams and charts).
The plain-text editors, initially line-oriented in the 1960s and early 1970s, became increasingly screen oriented in the late 1970s and early 1980s. They had one purpose: put text, essentially unformatted and with no concept of typography, into buffers with the least amount of effort. Examples of screen-oriented plain-text editors are vi and emacs. The appeal of such an editor is simplicity. A small amount of structure, such as auto-indent, is easily imposed, and these simple editors can be embedded in other applications.
In office situations, plain-text editors are rarely used outside of e-mail applications. Most office documents require at least some structure, such as memos and letters, and the structured editors handle these applications very well. Yet, the plain-text editors survive because of their simplicity and the usefulness of creating text with minimal layout structure.
However, there are situations where neither plain-text editors nor structured editors are suitable. These are situations where notes need to be taken quickly and/or where greater flexibility in creating and manipulating information is needed.
Examples of such situations are:
(1) Taking notes on a board in front of a meeting. This might be on a physical whiteboard, where everyone can see what's written. This might also take place on a large pen-based electronic display, such as a LiveBoard or a SmartBoard.
(2) Taking notes on a “blank sheet”, either in a meeting or working alone. The notes might be taken on paper with a pen or pencil. They might be taken on a pen-based electronic device. They might be taken on a laptop with a plain-text editor.
(3) Making notes on an existing document, either in a meeting or working alone. This might be done on paper with pencil or pen, or it could be done on an electronic device that can display a document and allow annotations to be made on the displayed document.
In these situations, a new kind of editor is preferred, which we call a “freeform editor”. A freeform editor allows “freeform graphic elements” (FGE(s)) to be freely created and moved around on a two-dimensional surface. By contrast, a plain-text editor is not freeform, because text cannot be put anywhere (such as in the margins or on top of other text). The most common kind of freeform editors are pen-based drawing programs that allows “digital ink” strokes to be created, thus simulating a whiteboard or notebook, which we call “scribble editors”. The FGE(s) of a scribble editor are the strokes. Other kinds of drawing systems that allow FGE(s) (such as pictures, icons, or test objects) to be moved around in an unconstrained way are also instances of freeform editors.
There are several useful things to be done with the freeform material (i.e. the FGE(s)) of a freeform editor.
(1) It would be useful to be able to structure the FGE(s). For example, the strokes in a scribble editor might want to be structured into words, lines of words, items in a list, groups of words as cells in a table, etc. An editor that allows such structuring of the FGE(s) is called a “structured freeform editor” (SFE). The capability that distinguishes a SFE from normal structured editors is that the user can freely move FGE(s) in a SFE, and the editor automatically recognizes the structures and preserves the structural relations as editing operations take place. An example of a SFE is the MeetingBoard program for the LiveBoard (a product of LiveWorks Inc., a former Xerox company).
(2) Another useful operation on freeform material would be to be able to transcribe non-textual FGE(s) into text, so that many textural operations such as searching and indexing may take place. In this regard, it would be desirable for transcription to be done automatically by handwriting recognition algorithms or manually by the user inputting the transcription or a mixture of both automatic and manual techniques.
(3) Finally, it would be useful to format freeform material (the FGE(s), their structure, and their transcriptions) to be input to a regular structured editor. For example, the scribbles made on a LiveBoard might be structured into a table structure and then formatted to be input into a Microsoft Word table.
The present invention proposes a new kind of structure freeform editor that allows freeform material to be imported from freeform editors or from scanned or photographic images, structured and restructured, manipulated and edited, new material to be created, transcribed, and exported in formats to be input into structured editors.